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GLOSSARY 

BIS 


BIS-CO 

CAT 


CD 

CTI 


cx 


CXIS 


CXIS-CO 


The  Baggage  Inspection  System  is  the  specific,  infonnal  name 
of  the  overall  EDS  utilizing  the  CTX  5000  X-ray  inspection 
subsystem.  The  BIS  is  designed  to  protect  civil  aviation 
consumers  and  commercial  interests  from  terrorists  and  other 
criminal  action  perpetrated  through  checked  baggage.  It  includes 
all  materials,  equipment,  facilities,  procedures,  training, 
manpower,  special  agreements,  logistics  support,  threat 
containment  and  disposal  provisions,  safety,  health,  and  technical 
data  necessary  to  install  and  operate. 

This  item  has  similar  elements  to  that  above  except  this  system  is 
configured  to  inspect  carry-on  baggage  using  the  CTX  5000. 

Computed  Axial  Tomography  X-ray  equipment  represents  a 
general  category  of  dynamic  X-ray  equipment  in  which  the  X-ray 
source(s)  and  collector(s)  revolve  about  the  baggage  taking 
multiple  cross-section  slices.  The  resulting  display  image  is  a  more 
detailed,  mass-density  mapping  of  X-ray  attenuation  in  each  voxel 
based  on  the  photoelectric  effect.  InVision's  CTX  5000  is  the  only 
member  of  this  category  under  evaluation  at  the  issuance  of  this 
TEMP. 

The  Connection  Device  is  a  universal  interface  linking  any  TIP 
system  to  any  CX  in  use. 

A  Combined  Test  Item  is  a  digitized  image  of  a  cluttered  bag 
with  a  threat  article(s).  Both  the  bag  with  its  threat  are  fictional 
and  digitally  characterized  images,  and  are  designed  to  work  in 
concert  with  CAT  and  CX  operation. 

Conventional  X-ray  equipment  represents  a  general  category  of 
static,  single  and  dual-energy  X-ray  equipment  which  all  generate 
a  qualitative  mass-density  mapping  of  X-ray.  Example:  EG&G 
Astrophysics  E-SCAN  X-ray  System. 

Conventional  X-Ray  Inspection  System  represents  the  overall 
inspection  system  using  CX  equipment  to  inspect  checked 
baggage.  It  includes  all  materials,  equipment,  facilities, 
procedures,  training,  manpower,  special  agreements,  logistics 
support,  threat  containment  and  disposal  provisions,  safety, 
health  and  technical  data  necessary  to  install  and  operate. 

This  system  has  similar  elements  to  that  above  except  this  system 
is  configured  to  inspect  carry-on  baggage  using  CX  equipment. 


VI 


AvSec  HF  TEMP 
20  June  1995 


EDS 

Fn 

lED 

Off-line 

On-line 

PD&I 

PETS 


Real-time 

SPEARS 

TIP 


The  Explosives  Detection  System  is  the  automatic  baggage 
screening  equipment  component  of  the  BIS.  An  EDS  is  a 
decision  support  system  that  delivers  alarm  and  cueing 
information  supporting  alarm  resolution  by  screeners. 

Fictional  Threat  Images  are  threat  images  that  can  be 
superimposed  upon  an  actual  bag  image  using  real-time 
projection  techniques  in  conjunction  with  on-line  CX  baggage 
screening  displays.  Only  the  threat  image  is  fictional,  the  bag 
image  is  not. 

An  Improvised  Explosive  Device  is  a  threat  object  or  article. 

Off-line  refers  to  a  training  activity  typically  away  from  the 
screener’s  checkpoint  job  site  or  while  the  screeners  are  not 
performing  their  job. 

On-line  refers  to  activity  that  is  simultaneous  with  the  screeners’ 
performance  on-the-job  at  the  checkpoint. 

Procurement,  Distribution,  and  Installation  are  a  logical 
grouping  of  specific  preparatory  tasks  for  the  Airport 
Demonstrations. 

Proficiency  Enhancing  Techniques  for  Screeners  encompass  all 
prototype  techniques  in  this  TEMP  that  seek  to  enhance  screener 
proficiency.  PETS  include  screener  selection  test  batteries, 
interface  improvements  for  the  CAT  system,  SPEARS  Off-line 
Training  Modules  for  CX  and  CAT,  and  SPEARS  On-line 
Testing/Training  Modules  for  CX  and  CAT. 

Real-time  refers  to  a  standard  rate  at  which  events  actually  occur. 
Real-time  is  in  contrast  to  off-line  training  and/or  testing. 

Screener  Proficiency  Evaluation  And  Reporting  Systran 
consists  of  an  Off-Line  Training  Module  and  On-Line 
Testing/Training  Module.  The  latter  module  is  also  known  as 
TIP  and  includes  a  performance  monitoring  capability. 

Threat  Image  Projection  is  the  on-line  component  of  SPEARS 
that  provides  real-time  training  and  testing  designed  to  work  in 
concert  with  CX  operation.  TIP  uses  either  CTI  or  FTI. 
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FAA  Civil  Aviation  Security 
Additional  Evaluation  Issue  and  Criteria 
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1.0  INTRODUCTION 

This  document  describes  the  Research,  Development,  Test  and  Evaluation  (RDT&E)  of  four 
Proficiency  Enhancement  Techniques  for  Screeners  (PETS)  as  well  as  one  custom 
interface,  called  the  Connection  Device  (CD),  in  support  of  Civil  Aviation  Security  (ACS) 
Research  Project  Directive  (RPD)  #127.  The  test  and  evaluation  (T&E)  portion  of  this 
program  encompasses  at  least  20  unique  activities  that  culminate  in  Airport  Demonstrations  at 
Category  X  airports. 

PETS  seeks  to  examine,  enhance,  and  improve  screener  performance  on  two  types  of  X-ray 
baggage  inspection  hosts:  Conventional  X-ray  (CX)  and  Computed  Axial  Tomography  (CAT) 
X-ray  explosive  detection  systems  (EDS).  PETS  consists  of  screener  selection  test  batteries, 
interface  improvements,  training,  and  performance  testing  modules.  The  latter  two  techniques 
are  called  Screener  Proficiency  Evaluation  And  Reporting  System  (SPEARS)  devices. 

The  results  of  the  test  activities  described  by  this  Test  and  Evaluation  Master  Plan  (TEMP) 
will  serve  to  support  deployment  and  regulation  decisions  by  defining  a  comprehensive  plan  to 
address,  resolve,  and  report  critical  issues. 

Section  1  of  this  document  covers  the  background,  purpose,  and  scope  of  the  RDT&E  program 
with  emphasis  on  operational  test  and  evaluation  (OT&E)  including  a  brief  summary  of  key 
activities.  Section  2  describes  the  key  systems  under  test  as  well  as  test  issues.  Section  3 
describes  the  program  management  structure,  funding,  and  responsibilities.  Section  4  defines 
the  key  test  activities  to  resolve  the  critical  parameters  and  issues  identified  in  Section  2,  and 
Section  5  lists  referenced  documents. 


1.1  Background 

Current  systems  do  not  meet  the  threat.  Currently,  the  most  effective  defense  against 
bombs  in  carry-on  and  checked  baggage  is  hand  inspection  combined  with  conventional  X-ray 
inspection,  which  is  labor  intensive  and  costly.  CX  systems  and  their  current  training 
programs,  which  rely  on  quick  interpretation  of  complex  two  dimensional  images  of  baggage, 
may  not  be  adequate  to  detect  artfully  concealed  explosives. 

Recent  actions  bring  focus.  As  a  result  of  studies  and  terrorist  actions,  the  United  States 
Congress  passed  the  Aviation  Security  Improvement  Act  of  1990  (Public  Law  101-604), 
directing  and  empowering  the  Federal  Aviation  Administration  to  accelerate  its  research, 
development,  and  implementation  of  key  technologies  to  improve  civil  aviation  security.  The 
President’s  Commission  on  Aviation  Security  and  Terrorism  (reference  5.e)  and  the  General 
Accounting  Office  (references  5.c  and  5.d)  have  reviewed  FAA  progress  in  this  area  on 
numerous  occasions,  and  have  recommended  greater  focus  on  human  factors  and  training  to 
leverage  and  complement  advanced  technology. 
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Rnhanring  the  human  element.  Screeners  are  critical  participants  in  aviation  security. 
Though  new  equipment  will  automate  some  aspects  of  the  baggage  inspection  process  and 
provide  the  screener  with  better  information,  overall  inspection  effectiveness  will  remain 
dependent  upon  the  proficiency  of  the  screener.  Similarly,  enhancing  the  proficiency  of  the 
screener  may  improve  the  effectiveness  of  current  inspection  systems.  In  June  1991,  the 
Aviation  Security  Human  Factors  Program  was  created  to  augment  the  Aviation  Security 
Research  and  Development  (R&D)  Division’s  detection  equipment  development  initiatives  by 
focusing  its  efforts  on  improving  the  performance  of  the  equipment  operators  referred 
throughout  this  document  as  screeners. 

ACS  Direction.  Direction  stems  from  ACS  Mission  Need  Statement  (MNS)  #163  seeking  to: 

•  “Maximize  the  capability  of  civil  aviation  security  personnel  to  detect  threats  and  deter 
terrorist  acts  and 

•  Optimize  human  performance  through  advanced  human  factors  (HF)  engineering  to 
improve  methods  of  screener  selection,  interface  improvements,  training,  performance 
testing,  and  motivation.  ” 


1.2  Purpose 

The  overall  objective  of  this  test  and  evaluation  effort  is  to  assess  the  operational  effectiveness 
of  a  suite  of  prototype  systems  that  seek  to  enhance  the  proficiency  of  the  human  screener  for 
CX  and  CAT  host  systems.  The  five  specific  research  objectives  are  as  follows: 

a.  Collect  and  analyze  empirical  data  on  screener  performance  using  X-ray  systems. 

b.  Obtain  and  assess  commercial-off-the-shelf  (COTS)  devices  for  a  prototype  CX 
SPEARS. 

c.  Develop  a  CAT  SPEARS  prototype. 

d.  Develop  new  processes,  procedures  and  technologies  for  improving  screener 
performance. 

e.  Develop  proficiency-based  taxonomy,  guidelines,  and  criteria  for  selection  and 
training. 


1.3  Scope 

To  accomplish  the  objectives  stated  in  section  1.2,  this  TEMP  defines  and  outlines  36  unique 
activities  that  encompass  the  full  research,  development,  test  and  evaluation  spectrum  for  each 
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of  the  following  four  PETS  as  well  as  for  a  Connection  Device  (CD)  to  support  threat  image 
projection.  These  techniques,  and  the  device  to  be  developed  and  tested,  are  listed  in  section 
1.3.1.  Section  1.3.2  defines  the  key  activities  of  a  general  project  analysis  followed  by  a 
summary  of  specific  activities  in  section  1.3.3  and  a  description  of  how  they  map  to  the 
research  objectives  in  section  1.3.4. 


1.3.1  Connection  Device  CCD)  and  Enhancement  Techniques 

This  project  primarily  seeks  to  improve  screener  productivity  through  the  use  of  the  following 
new  techniques  and  a  CD  resulting  in  potentially  eight  different  test  items.  The  first  two  and 
last  two  items  of  this  list  (a,  b,  d,  and  e)  constitute  PETS,  and  the  third  item  (c)  is  enabling 
equipment  for  the  SPEARS  On-line  Testing/Training  Module,  also  known  as  the  Threat  Image 
Projection  (TIP)  System. 

a.  Screener  Selection  Test  Batteries  for  both  CX  and  CAT  systems; 

b.  Interface  Improvements  for  the  CAT  system; 

c.  £D  for  CX  systems; 

d.  SPEARS  Off-Line  Training  Modules  for  both  CX  and  CAT  systems;  and 

e.  SPEARS  On-Line  Testing/Training  Modules  (TIP)  for  both  CX  and  CAT  systems. 


1.3.2  Key  Activities 

The  RDT&E  program  involving  these  developmental  items  (Section  1.3.1)  will  consist  of  four 
distinct  activity  categories.  The  latter  three  represent  the  major  test,  evaluation,  and  support 
activities.  Their  scope  is  defined  below: 


1.3.2.1  R&D:  Research  and  Development 

This  activity  includes  concept  exploration,  requirements  analysis,  specification  preparation,  issue 
identification  and  resolution,  engineering  design,  and  prototype  assembly. 


1.3.2. 2  DT&E:  Development  Test  and  Evaluation 

This  quality  assurance  activity  tests  for  compliance  to  technical  functional  requirements  as 
specified  and/or  limited  operational  effectiveness  in  a  controlled,  benign  environment  at  the 
vendor’s  facilities.  Only  FAA-witnessed  or  formal  DT&E  activities  are  indicated  on  the  table. 
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This  activity  involves  the  purchase,  distribution,  and  installation  of  test  articles  at  Category  X 
airports  to  support  OT&E. 

1.3.2.4  OT&E:  Operational  Test  and  Evaluation 

This  testing,  which  is  also  referred  to  as  the  Airport  Demonstrations,  has  two  phases,  irdtial, 
znd  follow-on,  which  support  the  operational  effectiveness  evaluation  of  selected  test  items. 

•  Initial  OT&E  (lOT&E)  is  the  first  test  phase  which  evaluates  the  test  item  under 
realistic  or  nearly  realistic  operational  conditions  at  operational  sites.  It  includes 
certified  screeners  and  FAA  test  baggage  and/or  passenger  baggage  in  real-time  as 
well  as  digitized  bag  images  off-line.  For  this  program,  both  CX  and  CAT  related 
systems  will  undergo  lOT&E,  but  only  CX  system  will  undergo  Baseline  Testing. 
Baseline  Tests  define  the  baseline  or  untreated  screener  performance  to  support 
comparative  analysis  and  evaluation.  AU  Baseline  Testing  will  generally  occur  at 
the  same  time  with  an  expected  test  duration  of  one  week  or  less  per  airport.  The 
conduct  of  CAT  SPEARS  lOT&E  will  be  coincident  with  the  Baggage  Inspection 
System  (BIS)-EDS  Demonstration. 

•  Follow-on  OT&E  (FOT&E)  wiU  consist  of  deployment  and  testing  of  the  test 
items  at  a  larger  diversity  of  selected  Category  X  airport  sites  for  CX  systems 
only. 


1.3.3  Summary  of  Kev  Activities 

As  a  whole,  this  Human  Factors  (HF)  project  encompasses  36  RDT&E  activities  as  shown  in 
Table  1.3-1.  The  unique  test  items,  derived  from  Section  1.3.1,  are  on  the  left  with  their 
scope  on  the  right.  The  performing  organization  is  identified  at  the  corresponding  junction. 
Table  1.3-2  spells  out  the  key  acronyms  identifying  the  performing  organizations  used  in  Table 
1.3-1.  Of  these  thirty-six  activities,  eight  pertain  to  R&D;  six  to  DT&E;  eight  to  PD&I;  and 
fourteen  to  OT&E. 

The  Host  column  shows  which  type  of  X-ray  inspection  system  the  items  wiU  be  designed  to 
support.  At  present,  the  specific  host  from  the  CX  category  is  EG&G's  E-SCAN  X-ray 
inspection  system,  and  the  specific  host  from  the  CAT  category  is  Imatron-InVision's  CTX 
5000  X-ray  inspection  system. 

The  darkly  shaded  areas  in  the  table  show  activities  completed  as  of  the  publication  of  this 
TEMP,  while  the  lightly  shaded  areas  represent  the  test  and  evaluation  focus  activities.  The 
activities  in  the  table  that  are  blank  are  either  not  to  be  performed  or  are  conducted  without 
formal  FAA  witness. 
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Table  1.3-1  Summary  of  Key  Activities 


Key  Activities 

iii 

ilii 

I 

Follow-on 

OT&E 

_ m _ 

Q 

D 

I 

FAA/TBD  I 

1 

Baseline 

Testing 

1 

FAA 

FAA 

1 

Initial 

OT&E 

ERAU 

FAA 

illlllllill 

FAA 

pK 

Air  Carriers 

FAA 

Air  Carriers 

PD&I 

FAA 

FAA 

TBD 

TBD 

FAA 

TBD 

FAA 

FAA 

DT&E 

ERAU 

ERAU 

S  .2 

1  ^ 

TBD 

InVision 

InVision 

R&D 

ERAU 

ERAU 

LLNL  and  InVision 

TBD 

lo 

■m 

Ills 

ill 

sp? 

InVision 

0 

1 

InVision 

Host 

CX 

CAT 

CAT 

CX 

CX 

CAT 

CX 

CAT 

Test  Item 

Screener  Selection  Test  Batteiy 
Screener  Selection  Test  Battery 

Interface  Improvements 

Connection  Devices 

SPEARS  Off-line  Training  Module 

SPEARS  Off-line  Training  Module 

SPEARS  On-line  Test  /  Training 

SPEARS  On-line  Test  /  Training 

1  ^ 

^  CS 

CO 

VO 
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1.3.4  Ob|ectives-to-Activities  Mapping 

The  relationship  of  the  five  research  objectives  listed  in  Section  1.2  (a  thru  e)  to  the  36 
activities  shown  in  Table  1.3-1  is  revealed  by  an  overlay  as  shown  in  Table  1.3-3,  Objectives- 
to-Activities  Mapping.  Table  1.3-4  displays  Section  1.2  (a  thru  e)  as  a  key  to  Table  1.3-3. 


Table  1.3-3  Objectives-to- Activities  Mapping 


a 

Test  Item 

Host 

Key  Activities 

R&D 

DT&E 

PD&I 

OT&E 

Initial 

OT&E 

.  Testily 

Eolkwp* 

on 

OT&E 

1 

Screener  Selection  Test  Battery 

CX 

M,e 

a,d,e 

a,d 

a,d 

mMm 

2 

Screener  Selection  Test  Battery 

CAT 

■PM 

a,d,e 

Bl 

iilili 

HHI 

3 

Interface  Improvements 

CAT 

a,d 

a,d 

a,d 

a,d 

mil 

4 

Connection  Devices 

CX 

a,d 

a,d 

HBI 

a,d[. 

■■■ 

illi  iflill 

5 

SPEARS  Off-line  Training  Module 

CX 

11%^ 

a,b,d 

a,b,d  i 

a,b4 . 

6 

SPEARS  Off-line  Training  Module 

CAT 

a,b,c,d 

iiiil 

a,b,d 

a,b,d 

imi 

imi 

7 

SPEARS  On-line  Test  /  Training 

CX 

mm 

.  a,b,d 

8 

SPEARS  On-line  Test  /  Training 

CAT 

a,b,c,d 

BGBI 

a,b,d 

— 

mi 

Table  1.3-4  Key  to  Table  1.3-3 

a.  Collect  and  analyze  empirical  data  on  screener  performance  using  X-ray  systems. 

b.  Obtain  and  assess  commercial-off-the-shelf  (COTS)  devices  for  a  prototype  CX  SPEARS. 
C,  Develop  a  CAT  SPEARS  prototype. 

d.  Develop  new  processes,  procedures  and  technologies  for  improving  screener  performance. 

e.  Develop  proficiency-based  taxonomy,  guidelines,  and  criteria  for  selection  and  training. 
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2.0  SYSTEM  DESCRIPnONS 

The  following  sections  describe  PETS  and  CD  characteristics,  interfaces,  and  critical 
parameters. 

2.1  Systems  Under  Test 

The  list  of  PETS  and  CD  in  section  1.3.1  expands  to  eight  unique  test  items  as  listed  in  Table 
2.1-1  with  its  developer  and  system  name.  Many  are  currently  unnamed.  The  right  most 
column,  displaying  refers  to  the  paragraphs  to  follow  that  describe  each  of  the  test  items. 


Table  2.1-1  Systems  Under  Test 


i 

TEST  ITEMS 

DEVELOPER 

SYSTEM  NAME 

D 

Screener  Selection  Test  Battery  for  CX 

ERAU 

TBD 

2.1.1 

Screener  Selection  Test  Battery  for  CAT 

ERAU 

TBD 

2.1.1 

3 

Interface  Improvements  for  CAT 

LLNL  -  InVision 

TBD 

BQi 

4 

Connection  Devices  for  CX 

TBD 

TBD 

2.1.3 

5„„ 

SPEARS  Off-line  Training  Module  for  CX 

SPIL  /  EG&G 

Safe  Passage  / 
TnT 

2. 1.4.1 

6 

SPEARS  Off-line  Training  Module  for  CAT 

InVision/  SPIL 

TBD 

2. 1.4.1 

7 

SPEARS  On-line  Testing/Training  Module  for  CX 

EG&G 

TnT 

SPEARS  On-line  Testing/Training  Module  for  CAT 

InVision 

TBD 

2. 1.4.2 

2.1.1  Screener  Selection  Test  Batteries 

2.1. 1.1  Description 

This  technique  seeks  a  predictor  of  CX  and  CAT  X-ray  systems  screener  performance 
effectiveness.  For  example,  among  various  candidate  predictors,  ERAU  is  measuring 
screeners  on  their  ability  to  discern  complex,  graphical  abstraction,  such  as  hidden  shapes. 
The  end  products  of  the  RDT&E  will  include: 

•  A  prototype  test  battery  to  include  specifications  for  the  hardware  and  software, 
protocols  and  administration  procedures  as  well  as  scoring  routines  for  generating 
selection  decisions. 

•  A  set  of  guidelines  and  recommendations  for  screener  selection. 
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ERAU,  under  an  FAA  Aviation  Grant,  is  developing  a  test  battery  for  CAT  and  CX  systems. 
In  general,  the  development  of  the  test  battery  involves  the  performance  of  the  following  tasks 
by  Dr,  Gibb  and  Dr.  Garland. 

•  Predictor  Test  Tool.  ERAU  will  design,  develop,  or  use  existing  predictor  tests  that 
may  have  a  relationship  to  screener  performance  and  that  can  be  evaluated  through  a 
predictive  validation  study. 

•  On  the  Job  Performance  Measure.  Threat  Image  Projection  performance  measures 
will  be  used  for  predictive  validity  and  reliability. 

•  Tool  Validation.  ERAU  will  conduct  a  validation  effort  in  operational  environments 
to  determine  the  relationships  among  the  possible  predictor  variables  and 
performance  criteria  selected  from  the  X-ray  technology.  TIP  criteria  used  shall  be 
indicative  of  success  or  failure  at  detecting  targets.  End  products  will  include  a 
prototype,  hardware  and  software  specifications,  and  supporting  statistical  analysis. 

2.1.2  CTX  5000  Interface  Improvements 
2.1.2.1  Description 

Interface  improvements  will  include  job  aids  and  computer-human  interaction  such  as 
information  entry  and  feedback  as  well  as  enhanced  displays  and  controls  for  the  CTX  5000. 


2. 1.2.2  Development  Approach 

Dr.  T.  Beimett  of  LLNL  will  conduct  a  design  audit  of  the  CTX  5000  interface.  The  resulting 
critical  human  factors  design  issues  will  be  resolved  by  In  Vision  and  tested  by  LLNL. 


2.1.3  Connection  Device 
2.1.3. 1  Description 

The  Connection  Device  is  a  universal  TIP  interface  device  or  “Black  Box”  to  permit  various 
versions  of  TIP  systems  to  operate  on  a  variety  of  CX  systems.  It  will  be  developed  by  one  or 
more  organizations  to  be  determined  and  will  be  based  in  part  on  studies  to  be  conducted  by 
Lincoln  Laboratories  at  Hanscom  AFB. 
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2. 1.3.2  Development  Approach 

An  organiTation  to  be  determined  will  survey  existing  CX  systems  and  develop  preliminary 
hardware  and  software  requirements  to  link  TIP  devices  to  existing  CXs. 

2.1.4  SPEARS 

The  function  of  SPEARS  is  to  perform  initial  and  recurrent  training,  and  to  monitor  screener 
performance.  SPEARS  consists  of  two  parts,  an  off-line  training  module  and  an  on-line  testing 
and  training  module.  There  are  also  types  of  SPEARS,  CX  SPEARS  and  CAT  SPEARS. 
These  systems  will  comply  with  the  SPEARS  functional  requirements  specification  (reference 

5.b). 

2. 1.4.1  SPEARS  Off-line  Training  Module 

2.1.4.1.1  Description 

This  off-line  trainer  provides  computer-based  training  (CBT)  that  is  not  coincident  with 
screener  operations.  In  general,  the  module  seeks  to  develop  and  maintain  threat  detection 
proficiency  in  a  variety  of  training  techniques. 


2.1.4.1.2  Development  Approach 

SPIL  and  EG&G  have  already  developed  their  systems  for  CX  systems.  SPIL  developed  the 
Vocation  Station  Safe  Passage  System  40  and  EG&G’s  TnT  operates  as  an  off-line  training 
system.  AAR-510  recently  completed  its  OT&E  of  these  items  and  identified  failures  to  meet 
critical  issues  which  need  to  be  addressed.  The  SPEARS  off-line  training  module  for  the  CAT 
will  be  developed  by  In  Vision  and  SPEL. 


2. 1.4.2  SPEARS  On-Line  Testing/Training  Module  (TIPI 

2. 1.4.2. 1  Description 

This  on-line  trainer  and  tester  permits  the  assessment  of  screener  proficiency  and  vigilance, 
and  incidentally  trains  screeners  to  better  detect  Improvised  Explosive  Devices  (lEDs).  It  does 
this  by  two  different  methods,  Fictional  Threat  Images  (FTI)  and  Combined  Threat  Items 
(CTI).  FTI  uses  superimposition,  electronically  overlaying  fictional  images  of  threat  objects 
onto  CX  images  of  actual  passenger  baggage,  unbeknownst  to  the  screener.  The  image 
appears  on  the  X-ray  monitor  as  if  a  threat  object  actually  exists  within  the  passenger’s  bag. 
Screeners  are  then  provided  feedback  on  their  ability  to  detect  the  FTI.  The  CTI  is  a 
prefabricated  image  of  a  threat  bag.  The  CTI  can  be  electronically  inserted  onto  the  CAT  or 
CX  display  monitors  for  the  same  purposes  as  the  FTI. 
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The  purpose  of  the  TIP  is  twofold.  It  exposes  screeners  to  threat  images  on  a  routine  basis  to 
enable  them  to  become  more  adept  at  recognizing  threats.  In  addition,  it  enhances  screens 
vigilance  by  unpredictably  inserting  fictional  threats  throughout  the  course  of  their  duty  cycle. 


2.1.4.2.2  Development  Approach 

TIP  exists  for  the  CX  host,  called  EG&G’s  Astrophysics  Linescan  Testing  and  Training  (TnT) 
system,  but  needs  to  be  developed  by  In  Vision  for  the  CAT  system.  Both  systems  will  be 
evaluated  based  on  requirements  published  in  the  SPEARS:  Human  Factors  Functional 
Requirements  and  Procedures  Report  (reference  5.b). 

•  EG&G  Astrophysics  Linescan®  TnT  System  The  TnT  was  developed  by  EG&G 
Astrophysics  to  train  airport  baggage  screeners  with  Linescan  X-ray  equipment. 

•  Imatron  Federal  Svstems-In Vision.  Inc.  CTX  5000  Training  System  Imatron-In Vision 
is  developing  a  TIP  for  their  CAT  system  based  on  the  requirements  in  reference 
5.b. 


2.2  System  Interfaces 

Each  of  the  14  test  plans  as  listed  in  Table  3.3-1  when  completed  wiU  describe  the  system-to- 
system  interfaces  where  appropriate. 


2.3  Critical  Parameters 

2.3.1  Critical  Performance  Parameters 

Critical  performance  parameters  are  addressed  by  issues  discussed  in  section  2.3.2. 


2.3.2  Critical  Issues  and  Criteria 

Critical  issues  are  primary  questions  regarding  operational,  technical,  or  support  capabilities 
that  must  be  answered  about  each  system  under  test  before  the  system’s  overall  value  or 
effectiveness  can  be  estimated.  Criteria  are  the  standards  by  which  the  issues  are  addressed. 
Measures  of  Performance  will  be  defined  in  each  individual  test  plan  for  each  criterion. 
Thresholds  embedded  in  some  of  the  criteria  are  the  minimum  acc^table  levels  of 
performance  required  by  a  test  article  to  perform  its  intended  mission. 

Table  2. 3.2-1,  on  the  next  page,  summarizes  the  top  15  Critical  Operational  Issues  and 
Criteria  (COIC),  23  Critical  Technical  Issues  and  Criteria  (CTIC),  and  one  Additional 
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Evaluation  Issue  and  Criteria  (ABIC).  This  Issue  Summary  Table  is  an  index  for  the  following 
tables  (2.3.2-2  thru  2.3.2.13)  which  define  the  issues  and  criteria  for  each  technique/device. 
Each  of  these  three  issue/criteria  sets  (COIC,  CTIC,  and  AEIC)  is  sequentially  numbered  and 
shown  in  the  Issue  column  indicating  the  numbered  series  relevant  to  each  technique  and 
device. 


Table  2.3.2-1  Issue  Summary  Table 


BETS  and  CD 

Issues 

iiiiiiii 

CX  and  CAT  Screener  Selection  Test  Batteries 

COIC  (1  -  2) 

2.3.2-2 

Interface  Improvements  for  CAT 

COIC  (3) 

2.3.2-3 

Connection  Device 

COIC  (4) 

2.3.2-4 

CX-  SPEARS  Off-Line  Training  Module 

COIC  (5  -  6) 

2.3.2-5 

CTIC  (1  -  7) 

2.32-6 

AEIC  (1) 

2.3.2-7 

CAT-SPEARS  Off-Line  Training  Module 

COIC  (7  -  9) 

2.3.2-8 

CTIC  (8-13) 

2.3.2-9 

CX-SPEARS  On-Line  Testing/Training  Module 

COIC  (10  -  12) 

2.3.2-10 

CTIC  (14  -  19) 

2.3.2-11 

CAT-  SPEARS  On-Line  Testing/Training  Module 

COIC  (13  -  15) 

2.3.2-12 

CTIC  (20  -  23) 

2.3.2-13 

Table  2.3.2-2  COIC:  CX  and  CAT  Screener  Selection  Test  Batteries 


Issue  if 

Title 

Description 

Criteria 

COIC-Ol 

CX-Screener 
Selection  Test 
Battery  - 
Criterion-related 
validity 

Can  a  battery  of  tests  be  developed 
which  can  be  used  to  validly  and 
reliably  predict  screener  performance 
with  CX? 

•  Screener  performance  on  selection 
batteiy  tests  is  significantly 
correlated  with  on-the-job 
performance  measured  by  detection 
of  FTIs  and/or  CTIs. 

COIC-02 

CAT-Screener 

Can  a  battery  of  tests  be  developed 

•  Same  as  above  except  only  with 

Selection  Test 

which  can  be  used  to  validly  and 

cns. 

Battery  - 

reliably  predict  screener  performance 

Criterion-related 

with  CAT? 

validity 
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Table  2.3.2-3  COIC:  Interface  Improvements  for  CAT 


Issue  # 

Title 

Description 

Criteria 

COIC-03 

CAT  Interface 
Improvement  - 
Design  Impact 

Are  there  any  HF  design  features  or 
other  system  characteristics  that 
adversely  affect  the  ability  of 
screeners  to  operate  the  CAT 
SPEARS  effectively  and  safely  in 
operational  and  training  settings? 

•  Investigative  in  nature. 

Table  2.3.2-4  COIC:  CX-TIP  Connection  Device 


Issue  0 

Title 

Description 

Criteria 

COIC-04 

CX-TIP  CD  - 
Feasibility 

Is  it  feasible  to  develop  a  device 
that  is  capable  of  connecting  various 
FIP  systems  to  various  types  of 

X-ray  machines? 

•  No  degradation  in  function  listed  in 
Table  3.3.2-10. 

Table  2.3.2-5 

COIC:  CX  -  SPEARS  Off-Une  Training  Module 

Issue  # 

Title 

Description 

Criteria 

COIC-05 

Effectiveness 

Does  the  training  enhance  screener 
capability? 

•  Screener  probability  of  detection 
(Pj)  significantly  increased  after 
training. 

•  Screener  d'  significantly  increased 
after  training 

•  Screener  probability  of  false  alarm 
(Pfa)  is  not  significantly  increased 
after  training. 

COIC-06 

Usability 

Are  there  any  software  or  hardware 
factors  or  procedural  aspects  that 
degrade  training  effectiveness? 

•  Investigative  in  nature. 
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Table  2.3.2-6  CTIC:  CX  -  SPEARS  Off-line  Training  Module 


Issue  # 

Title 

Description 

Criteria 

CTIC-Ol 

Image  Content 

Do  the  images  represent  the  range 
of  current  threats? 

•  300  different,  representative  CTIs 
are  available  with  representative 
cluttered  bags 

•  ^2  different  aspect  angles  for  each 
threat 

•  Degree  of  detection  difficulty  varies 

CTIC-02 

Customization 

Can  training  be  tailored  to 
individual  needs? 

•  Images  can  be  selected  for  each 
screener  as  a  function  of  threat 
category,  difficulty,  and  aspect 
angle  which  are  then  archived  along 
with  the  associated  outcome  (hit, 
miss,  false  alarm,  correct  rejection). 

CTIC-03 

Feedback 

Is  feedback  provided? 

•  Screener  is  immediately  informed 
on  whether  each  threat  object  was 
correctly  identified. 

CTIC-04 

Capability 

Summaries 

Are  training  reports  prepared? 

•  Training  reports  containing 
descriptive  statistics  and  readily 
understandable  interpretations  are 
automatically  generated  to 
summarize  records  of  the  threats 
presented  and  the  outcomes. 

CTIC-05 

Interoperability 

Can  the  equipment  communicate 
with  remote  computers? 

•  Additional  FTIs  and  CTIs  are 
received  from  remote  sites. 

•  The  equipment  will  be  able  to 
transmit  reports  to  remote  sites. 

CTIC-06 

Security 

Is  access  restricted? 

•  Only  authorized  screeners  and 
supervisors  can  access  certain 
aspects  of  the  system. 

CTIC-07 

Multimodal 

Is  instruction  provided  in  the  visual 
and/or  auditory  modalities? 

•  Contents  can  be  read  by  trainee 

•  Contents  can  be  read  to  trainee 

Table  2.3.2-7  AEIC:  CX  -  SPEARS  Off-line  Training  Module 


Issue# 

Title 

Description 

Criteria 

AEIC-Ol 

CX-Training 
Module  Reading 
Level 

What  reading  capability  is 
required  for  the  training? 

•  Investigative  in  nature. 
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Table  2.3.2-8  COIC:  CAT-SPEARS  Off-line  Training  Module 


Issue  # 

Title 

Description 

Criteria 

COIC-07 

Equipment 

Familiarization 

Does  the  training  enable  screeners 
to  operate  the  CTX  5000 
equipment  in  general  and  the 
alarm  resolution  interface  in 
particular? 

•  Investigative  in  nature. 

COIC-08 

Effectiveness 

Does  the  training  enhance  screener 
capability  to  detect  lEDs? 

•  Pa  is  significantly  increased 
after  training 

•  d'  is  significantly  increased 
after  training. 

•  Pfe  is  not  significantly 
increased  after  training 

COIC-09 

Usability 

Are  there  any  software  or 
hardware  factors  or  procedural 
aspects  that  degrade  training  or 
operational  effectiveness? 

•  Investigative  in  nature. 

Table  2.3.2-9  CTIC:  CAT-SPEARS  Off-line  Training  Module 


Issue  # 

Title 

Description 

Criteria 

cnc-08 

Image  Content 

Do  the  images  represent  the  range 
of  currently  lED  threats? 

•  300  different  CTIs  representing 

each  of  the  6  bulk  types  with 
lED  components  with 
representative  cluttered  bags 
are  available 

CTIC-09 

Customization 

Can  training  be  tailored  to 
individual  screeners  needs? 

•  Images  can  be  selected  for  each 
screener  as  a  ftmction  of 
explosive  type  which  is 
archived  along  with  the 
associated  outcome  (hit»  miss, 
false  alarm,  correct  rejection) 

cnc-10 

Feedback 

Is  feedback  provided? 

•  Screener  is  immediately 
informed  on  whether  each 
threat  object  is  correctly 
identified. 
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Table  2.3.2-9  CTIC:  CAT-SPEARS  Off-line  Training  Module  (cont.) 


Issue  # 

Title 

Description 

Criteria 

cnc-ii 

Capability 

Summaries 

Are  training  reports  prepared? 

•  Training  reports,  containing 
descriptive  statistics  and  readily 
xmderstandable  interpretations 
are  automatically  generated  to 
summarize  records  of  threats 
presented  and  the  outcomes. 

cnc-12 

Security 

Is  access  restricted? 

•  Only  authorized  screeners  and 
supervisors  access  certain 
aspects  of  the  system 

CTIC-13 

Multimodal 

Is  instruction  provided  in  the  visual 
and/or  auditory  modalities? 

•  Contents  can  be  read  by  trainee 

•  Contents  can  be  read  to  trainee 

Table  2.3.2-10  COIC:  CX-SPEARS  On-line  Testing/Training  Module 


Issue  # 

Title 

Description 

Criteria 

COIC-10 

Insertion 

Can  FTI  and  CTI  be  automatically  and 
unpredictably  inserted  on  the  interface 
display? 

•  The  FTI  and  CTI  appears  no 
different  that  a  real  image  of  a 
bag  that  is  currently  in  the  X- 
ray  system. 

COIC-11 

Vigilance 

Do  projected  images  enhance  screener 
capability? 

•  Projecting  fictional  images 
increases  P^. 

•  Projecting  fictional  images 
increases  d'. 

•  Projecting  fictional  images  does 
not  increase  P^. 

COIC-12 

Usability 

Are  there  any  software  or  hardware 
factors  or  procedural  aspects  that  degrade 
training  effectiveness? 

•  Investigative  in  nature. 
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Table  2.3.2-11  CTIC:  CX-SPEARS  On-line  Testing/Training  Module 


Issue# 

Title 

Description 

Criteria 

cnc-14 

Image  Content 

Do  the  images  represent  the  range  of 
current  threats? 

•  300  different,  representative  CTIs  and 
FTIs  are  available 

•  k2  different  aspect  angles  for  each 
threat 

•  Degree  of  detection  difficulty  varies 

cnc-15 

Customization 

Can  performance  monitoring  be 
tailored  to  individual  screener  needs? 

•  Images  can  be  selected  for  each 
screener  as  a  function  of  threat 
categoiy,  difficulty,  and  aspect  angle 
which  are  archived  along  with  the 
associated  outcome  (hit,  miss,  false 
alarm,  correct  rejection) 

•  FTIs  and  CTIs  are  automatically  or 
manually  selected  for  presentation 
based  on  time  of  day,  checkpoint 
activity,  or  screener  ID. 

cnc-16 

Feedback 

Is  feedback  provided? 

•  Screener  is  immediately  informed  on 
whether  each  threat  object  is  correctly 
identified. 

cnc-17 

Capability 

Summaries 

Are  performance  reports  prepared? 

•  Performance  reports,  containing 
descriptive  statistics  and  readily 

!  understandable  interpretations  are 

automatically  generated  to  summarize 
records  of  threats  presented  and  the 
outcomes. 

•  Supervisors  are  alerted  when  screener 
performance  falls  below  operational 
requirements  during  a  screening 
session. 

cnc-18 

Interoper¬ 

ability 

Can  the  equipment  communicate  with 
remote  computers? 

•  Additional  FTIs  and  CTIs  are 
received  from  remote  sites 

•  Equipment  is  able  to  transmit  reports 
to  remote  sites 

cnc-19 

Security 

Is  access  restricted? 

•  Only  authorized  screeners  and 

supervisors  can  access  certain  aspects 
of  the  system 
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Table  2.3.2-12  COIC:  CAT-SPEARS  On-line  Testing/Training  Module 


Ibsue  # 

Title 

Description 

Criteria 

COIC-13 

Insertion 

Can  CTI  be  automatically  and 
unpredictably  inserted  on  the  interface 
display? 

•  The  CTI  appears  no  different  that  a  real 
image  of  a  bag  that  is  currently  in  the 
system 

•  No  aspect  of  the  interface  reveals  to  the 
screener  the  use  of  a  CTI  in  place  of  a 
‘real’  bag  image 

COIC-14 

Vigilance 

Do  projected  images  enhance  screener 
capability? 

•  Projecting  fictional  images  increases 

•  Projecting  fictional  images  increases  d'. 

•  Projecting  fictional  images  does  not 
increase  P^. 

COIC-15 

Usability 

Are  there  any  software  or  hardware 
factors  or  procedural  aspects  that 
degrade  training  effectiveness? 

•  Investigative  in  nature. 

Table  2.3.2-13  CTIC:  CAT-SPEARS  On-line  Testing/Training  Module 


Issue  # 

Title 

Description 

Criteria 

CTIC-20 

Feedback 

Is  feedback  provided? 

•  Screener  is  immediately  informed  on 
whether  each  threat  object  is  correctly 
identified. 

CTIC-21 

Security 

Is  access  restricted? 

•  Only  authorized  screeners  and  supervisors 
can  access  certain  aspects  of  the  system 

cnc-22 

Image  Content 

Do  the  images  represent  the  range 
of  current  threats? 

•  300  different  CTIs  representing  each  of 
the  6  explosive  types  with  lED 
components  in  representative  cluttered 
bags  are  available 

•  These  CTIs  differ  from  off-line  training 

CTIC-23 

Customization 

Can  performance  monitoring  be 
tailored  to  individual  needs? 

•  Images  can  be  selected  for  each  screener  as 
a  function  of  explosive  type  which  is 
archived  along  with  the  associated 
outcome  (hit,  miss,  false  alarm,  correct 
rejection) 

•  CTIs  are  automatically  or  manually 
selected  for  presentation  based  on  time  of 
day,  checkpoint  activity  or  screener  ID 
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2.3.3  Minimum  Operational  Interference 
The  test  and  evaluation  activities  will  not: 

•  Compromise  security;  nor 

•  Cause  undue  hardship  for  air  carriers  or  significant  interruptions  to  flow  of  personnel 
or  baggage  within  the  airport. 


2.4  Completion  Criteria 

Each  of  the  evaluations  can  be  completed  when  the  criteria  in  Table  3.4-1,  Activity 
Completion  Criteria  Checklist  have  been  fulfilled. 


Table  2.4-1  Activity  Completion  Criteria  Checklist 


CRITERIA 

1 .  System  acquisition,  installation,  operation,  and  maintenance  are  accurately  and 
completely  documented. 

2.  Manpower/staffing  is  in  place. 

3.  All  personnel  are  fully  trained. 

4.  Equipment  and  computer  programs  are  installed,  tested,  and  are  fully 
operational 

5.  Procedures  are  fully  documented  and  verified. 

6.  The  laboratory  or  test  site  is  configured  properly  and  incorporates  all  necessary 
facilities  and  support. 

7.  All  safety,  maintainability,  and  operability  considerations  have  been  identified 
and  documented. 
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3.0  PROGRAM  MANAGEMENT 

The  program  sponsor  is  ACS-1  and  overall  program  management  is  performed  by  the  Aviation 
Security  Human  Factors  Program  Manager.  The  primary  participants  are  In  Vision,  LLNL,  air 
carriers,  ERAU,  a  TIP  Connection  Device  Vendor,  and  support  contractors  including  Galaxy 
Scientific  Corporation  (GSC)  and  DCS,  Inc.  InVision  and  ERAU  will  be  in  a  Cooperative 
Research  Agreement  with  the  FAA.  LLNL  operates  under  an  interagency  agreement  with 
AAR-510.  The  organizational  relationships  are  shown  in  Figure  3-1.  The  shaded  boxes 
identify  the  focus  and  contents  of  this  HF  TEMP  under  the  direction  of  Dr.  J.  L.  Fobes. 


Figure  3-1.  Program  Management  Structure 
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3.1  Management 

This  section  describes  the  organizations,  roles,  and  responsibilities  for  the  R&D,  DT&E, 

PD&I  and  OT&E  activities  as  shown  in  Table  3.1-1. 


Table  3.1.1-1  RDT&E  Responsibilities 


Organization 

Responsibilities 

ACS/ACP 

•  Sponsors  Program 

•  Provides  requirements  and  overall  direction 

•  Approves  TEMP 

AAR-510 

•  Provides  overall  program  management 

•  Provides  funding 

•  Directs  and  witnesses  DT&E  of  CAT  SPEARS  Off-line  Training  Module  and 
SPEARS  On-line  Testing/Training  Module  (TIP) 

•  Witnesses  testing  of  CAT  Interface  Improvements  and  Connection  Device 

•  Directs  and  witnesses  OT&E  and  Airport  Demo  of  SPEARS  Off-line  Training 
Module 

•  Directs  and  witnesses  SPEARS  On-line  Testing/Training  Module 

•  Prepares  and  issues  specifications  and  purchase  orders  and  deploys  SPEARS  for 
both  CX  and  CAT  system 

•  Coordinates  planning  and  test  requirements  with  BIS  Demo 

Program  Manager  &  Test  Director 

•  Conducts  OT&E  for  CDs  and  CAT  Interface  Improvements 

TBD 

•  Develops  and  tests  Connection  Device 

ERAU 

•  Develops  and  evaluates  screener  selection  test  batteries  for  both  CX  and  CAT 
systems 

•  Conducts  OT&E  Screener  Selection  Test  Batteries  for  both  CX  and  CAT 
systems 

InVision 

•  Executes  design  changes,  conducts  lab  testing,  and  implements  changes 

•  Develops  and  tests  the  TIP  for  the  CAT  system 

•  Develops  and  tests  the  SPEARS  Off-line  Training  Module  for  the  CAT  system 
(with  SPIL) 

EG&G  /  SPIL 

•  Develops  and  provides  SPEARS  for  CX 

LLNL 

•  Conducts  human  factors  audit,  analyzes  interface  requirements,  and 
recommends  improvements  for  the  CAT 

•  Participates  in  CAT  DT&E  activities 

Air  Carriers 

•  Receives,  coordinates,  and  installs  SPEARS  in  airports 

•  Conducts  OT&E  and  Airport  Demo  of  both  SPEARS  test  items  for  CX  and 

CAT  systems 
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3.2  Test  Resources 
3.2.1  Funding 

Funding  and  programming  to  accomplish  key  taskings  is  shown  in  Table  3.2. 1-1  Funding 
Profiles.  The  use  of  the  term  “Grant”  reflects  a  general  contractual  category.  The  type  of 
grant  to  be  used  is  a  Cooperative  Agreement. 

Table  3.2. 1-1  Funding  Profiles 


1^9 

Progranuning 

FY96 

FY97 

FY98 

Row  Total 

■■ 

A 

InVision  Grant 

$ 

$ 

$ 

$ 

B 

ERAU  Grant 

$ 

$ 

$ 

$ 

C 

LLNL  Interagency  Agreement 

$ 

$ 

$ 

$ 

D 

Connection  Device 

$ 

$ 

$ 

$ 

E 

AAR-510  Support  Contracts 

$ 

$ 

$ 

$ 

F 

CX  SPEARS  CBT 

$ 

$ 

$ 

$ 

G 

CX  SPEARS  TIP 

$ 

$ 

$ 

$ 

Column  Total 


$ 


$ 


$ 


3.2.2  Personnel 

The  following  is  a  list  of  the  categories  of  personnel  required  to  conduct  testing  activities 
shown  in  Table  3. 2. 2-1  Table  3. 2. 2-2  shows  their  primary  duties  and  responsibilities.  The 
term  vendor  is  used  to  denote  any  of  the  fundees  to  include  InVision,  ERAU,  LLNL,  EG&G, 
or  others  to  be  determined.  The  term  AAR-510  is  used  to  include  this  organization’s  technical 
support  contractors,  GSC  and  DCS,  Inc. 


Table  3.2.2-1  Site  Personnel  Support 


Title 

DT&E 

OT&E 

Program  Sponsor 

ACS-1 

ACS-l 

Program  Manager 

AAR-510 

AAR-510 

Test  Engineer(s) 

AAR-510A^endor 

AAR-5 10/ Air  Carrier 

AAR-510/TBD 

Air  Carrier 

Baggage  Handlers 

TBD 

Air  Carrier 

Logistics  Coordinator 

AAR-510 

AAR-510 

Response  Force  Personnel 

N/A 

Airport  Authority 

Screeners 

TBD 

Air  Carrier 

Technicians 

Vendor 

Air  Carrier/Vendor 

NOTE:  N/A  means  “not  applicable 
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Table  3.2.2-2  Responsibilities  of  Key  Personnel 


Job  Title 

Duties/Responsibilities 

Program  Sponsor 

•  Sponsors  Research  Project  Directives  for  security  systems  research, 
engineering,  and  development 

•  Provides  security  system  requirements,  tasking,  and  overall  review 
of  AAR-500  efforts 

•  Reviews  AAR-500  funding  allocations 

Av  Sec  HF  Program 

•  Coordinates  fundiag  for  projects/activities 

Manager 

•  Provides  laboratory  facilities  for  PETS  testing 

AAR-510 

•  Coordinates  with  FAA  Headquarters,  Washington,  DC 

•  Has  overall  approval/authority  over  all  Human  Factors  including 

PETS  activities 

•  Coordinates  with  higher  level  FAA  and  airport  management  on 
airport  test  activities 

•  Manages  schedules  and  budgets  for  Human  Factors 

•  Reviews/approves  human  factors  PETS  activities  performed  by 
vendors,  grantees,  and/or  contractors 

•  Prepares  and  publishes  final  test  plans,  reports,  and  other 
documentation  on  Human  Factors  PETS 

•  Supervises/directs  test  team  members  in  conduct  of  laboratory  and 
field  site  tests 

•  Oversees  data  collection,  reduction,  evaluation,  and  reporting 
activities  of  test  team  members 

•  Ensures  adequate  manpower  resources  are  available  for 
training/testing 

Test  Engineer(s) 

•  Collects  training,  performance  proficiency,  and  interface  data  during 
testing 

•  Prepares  and  executes  test  plans/procedures 

•  Ensures  test  subjects  are  interviewed  or  complete  questionnaires  at 
completion  of  testing 

•  Ensures  data  collection  sheets  and  questionnaires  are  turned  in  at 
completion  of  testing 

•  Performs  test  data  reduction,  evaluation,  and  reporting  tasks 

•  Provides  guidance/assistance  to  screeners  during  conduct  of  tests  as 
required 

•  Briefs  Program  Manager  as  appropriate  on  any  problems  occurring 
with  the  testing 

Equipment  C)perator(s) 

•  Sets  up,  operates,  and  makes  minor  adjustments/repairs  to  equipment 
during  testing 

•  Prepares  and  reports  on  equipment  problems/discrepancies  during 
test 
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Table  3.2.2-2  Responsibilities  of  Key  Personnel  (Cont.) 


Job  Title 

Duties/Responsibilities 

Baggage  Handler(s) 

•  Transports  bags/test  articles  from  storage  facility  to  test  area  prior  to 
testing  and  returns  items  to  storage  after  testing 

•  Loads/removes  bags  from  conveyor  belt/x-ray  machine  at  direction  of 
Logistics  Coordinator 

Logistics  Coordinator 

•  Oversees  handling,  transporting,  storing,  and  coordination  of  test 
baggage 

•  Ensures  bags  are  returned  to  proper  source 

•  Ensures  operability  and  provides  test  support  equipment  (e,g.,  hand¬ 
held  radios  and  wands)  to  test  subjects 

•  Ensures  availability  of  and  escorts  test  subjects  to  test  site  or  training 
facility 

•  Arranges  for  test  simulants  and  configures  bags  with  lEDs 

•  Oversees  baggage  handler  activities 

Response  Force  Personnel 

•  Provides  support  during  PETS  testing  in  accordance  with  Alarm 
Resolution  procedures 

•  Participates  in  test  briefings/debriefings  as  required 

Certified  Screeners 

•  Serve  as  subjects  during  tests 

•  Participate  in  test  briefings/debriefings  as  required  by  Test  Director 

•  Participate  in  post  test  interviews  and  completes  questionnaires  as 
appropriate 

Technicians 

•  Install  and  checks-out  equipment/software  at  laboratory  or  test  site 

•  Troubleshoot,  repair,  and  replace  defective  hardware/software  as 
required 

•  Maintain  records  of  activities  and  briefs  Test  Engineer(s)  on 
equipment/software  repair/replacement  status  as  appropriate 

3.2.3  Contracting  Management 

Within  AAR-510,  Dr.  J.  L.  Fobes  provides  technical  oversight  responsibility  and  quality 
assurance  for  all  seven  HF  contract  activities  listed  in  Table  3.2. 1-1. 


3.2.4  Test  Articles  and  Tools 

The  PETS  will  be  hosted  on  and/or  evaluated  with  CX  and  CAT  X-ray  inspection  systems  as 
identified  earlier.  Standard  threat  articles  within  the  TIP  database  will  be  created  based  upon 
standard  test  articles  being  developed  for  the  BIS  Demo  for  comparative  analysis. 


3.2.5  Support  Equipment 

Support  equipment  will  be  defined  in  the  test  plans  for  each  individual  test. 


23 


AvSec  HF  TEMP 
20  June  1995 


3.3  Test  Data  and  Products 

Test  data  deliverables  are  listed  in  Table  3.3-1.  The  term  TIP  Module  is  used  to  mean 
Testing/Training  Module  throughout  this  table.  Item  #29  is  the  final  research  rqiort  that 
synthesizes  all  findings  and  addresses  the  intended  end  products  for  RPD  #127.  The  intended 
two  end  products  of  this  research  project  are: 

•  Final  SPEARS  system(s) 

•  Establishment  of  criteria  for  selection,  training,  proficiency,  and  assessment  for  CX 


and  CAT  screeners 


Table  3.3-1  Data  Items 


iill 

Data  Items 

From 

To 

Due 

Date 

DT&E  Plan,  Screener  Selection  Test  Battery  for  CX 


DT&E  Plan,  Screener  Selection  Test  Battery  for  CAT 


DT&E  Plan,  Interface  Improvements  for  CAT 


DT&E  Plan,  CD  for  CX-TIP 


DT&E  Plan,  SPEARS  Training  Module  for  CAT 


DT&E  Plan,  SPEARS  TIP  Module  for  CAT 


ERAU 


ERAU 


LLNL 


TBD 


InVision 


InVision 


AAR-510 


AAR-510 


AAR-510 


AAR-510 


AAR-510 


AAR-510 


DT&E  Report,  Screener  Selection  Test  Battery  for  CX 


DT&E  Report,  Screener  Selection  Test  Battery  for  CAT 


DT&E  Report,  Interface  Improvements  for  CAT 


DT&E  Report,  CD  for  CX-TIP 


DT&E  Report,  SPEARS  Training  Module  for  CAT 


DT&E  Report,  SPEARS  TIP  Module  for  CAT 


ERAU 


ERAU 


LLNL 


TBD 


InVision 


InVision 


AAR-510 


AAR-510 


AAR-510 


AAR-510 


AAR-510 


AAR-510 


OT&E  Airport  Report, 


OT&E  Airport  Report, 


OT&E  Airport  Report, 


OT&E  Airport  Report, 


OT&E  Airport  Report, 


OT&E  Airport  Report, 


OT&E  Airport  Report, 


OT&E  Airport  Report, 


Screener  Selection  Test  Battery  for  CX  I  ERAU  I  AAR-510 


Screener  Selection  Test  Battery  for  CAT  ERAU  AAR-510 


Interface  Improvements  for  CAT  LLNL  AAR-510 


CD  for  CX-TIP  TBD  AAR-510 


SPEARS  Training  Module  for  CX  TBD  AAR-510 


SPEARS  Training  Module  for  CAT  TBD  AAR-510 


SPEARS  TIP  Module  for  CX  TBD  AAR-510 


SPEARS  TIP  Module  for  CAT  TBD  AAR-510 


29  I  RPD  #127  Final  Research  Report 


OT&E  Airport  Plan,  Screener  Selection  Test  Battery  for  CX 

ERAU 

AAR-510 

OT&E  Airport  Plan,  Screener  Selection  Test  Battery  for  CAT 

ERAU 

AAR-510 

OT&E  Airport  Plan,  Interface  Improvements  for  CAT 

LLNL 

AAR-510 

OT&E  Airport  Plan,  CD  for  CX-TIP 

TBD 

AAR-510 

OT&E  Airport  Plan,  SPEARS  Training  Module  for  CX 

TBD 

AAR-510 

OT&E  Airport  Plan,  SPEARS  Training  Module  for  CAT 

TBD 

AAR-510 

OT&E  Airport  Plan,  SPEARS  TIP  Module  for  CX 

TBD 

AAR-510 

OT&E  Airport  Plan,  SPEARS  TIP  Module  for  CAT 

TBD 

AAR-510 
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3.4  Test  Configuration  Management 

Each  of  the  prototype  developers  will  provide  AAR-510  sufficient  evidence  to  indicate  that  the 
configuration  of  their  system  under  test  is  under  control  in  accordance  with  each  test  plan  and 
procedures.  Sound  configuration  practices  of  documented  baselines  and  modification 
management  will  be  used. 


4.0  TEST  PROGRAM  DESCRIPTION 


4.1  Previous  Testing 

4.1.1  Improvised  Explosive  Device  Screening  Systems  (TEBSSl  T&E 

OT&E  at  San  Francisco  International  Airport  in  July/August  1994  investigated  whether 
detection  performance  can  be  improved  through  computer-based  training  (reference  5.g).  This 
study  compared  the  effectiveness  of  enhanced  and  black  &  white  X-ray-based  lED  detection 
systems  in  detecting  simulated  lEDs  in  checked  and  carry-on  baggage.  The  findings  are  listed 
below: 

•  No  significant  difference  in  lED  detection  performance  was  exhibited 
between  the  enhanced  and  black  &  white  X-ray-based  lEDDSs. 

•  Screeners  were  better  at  detecting  lEDs  in  carry-on  baggage  than  in  checked 
baggage  based  on  Pj  measure. 

•  Screener  sensitivity  (d'),  but  not  Pd,  was  higher  in  carry-on  bags  than  in 
checked  bags  using  the  enhanced  display  subsystem. 

•  Training  resulted  in  improved  performance  (P^  and  d').  The  false  alarm  rate 
(Pfa)  and  screener  decision  criteria  (P)  did  not  change. 

•  No  significant  correlations  were  found  between  screener  characteristics  (color 
vision,  visual  acuity,  and  experience)  and  the  performance  measures  of  pre¬ 
training  and  post  training  Pj  and  d'. 


4.1.2  T&E  Report  for  Explosive  Detection  Device  fCTX  50001  Alarm  Resolution 

Drs.  James  Connelly  and  J.  L.  Fobes  directed  a  DT&E  of  screener  alarm  resolution 
performance  using  the  InVision  CTX  5000  at  InVision’s  Foster  City  laboratory,  August  10-11, 
1994  (reference  5.h).  Twenty-two  test  bags  were  configured  with  five  simulated  lED 
configurations  from  the  Modular  Bomb  Set  Kit  and  randomly  inserted  among  220  normal 
passenger  bags.  Four  screeners  (two  InVision  experts  and  two  ITS  novice  screeners  who 
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previously  participated  in  an  lEDSS  T&E)  each  viewed  the  set  of  test  bags.  The  findings  are 
listed  below: 

•  All  four  screeners  were  effective  at  identifying  threats  within  passenger  baggage  by 
accurately  resolving  the  CTX  5000  system  alarms,  while  the  false  alarm  rate  was 
moderate  for  all  subjects. 

•  The  expert  screeners  were  significantly  faster  at  making  decisions  resolving  alarms 
than  were  the  novices,  resulting  in  higher  baggage  throughput  rates. 

•  Screener  decision  time  was  significantly  shorter  on  identifying  test  (threat)  bags  than 
for  normal  (non-threat)  passenger  bags. 


4.2  Test  Activities 


This  section  lists  each  of  the  T&E  activities  that  encompass  DT&E  and  OT&E,  the  issues  they 
address  as  identified  in  Section  2.3.2,  and  who  is  responsible  for  their  conduct.  Table  4.2-1 
displays  six  indexed  DT&E  activities  and  eight  indexed  OT&E  activities  as  shaded  using  the 
tabular  form  from  Table  1.3-1. 


Table  4.2-1  Indexed  Test  Activities 


it 

Test  Items 

Host 

Key  Activities  | 

R&D 

DT&E 

PD&I 

OT&E 

1 

Screener  Selection  Test  Battery 

cx 

1 

1 

2 

Screener  Selection  Test  Battery 

CAT 

iiiiiiiiii 

2 

3 

CAT  Interface  Improvements 

CAT 

iiiiiiiiil 

3 

4 

Connection  Devices 

CX 

llliM 

4 

5 

SPEARS  Off-line  Training  Module 

CX 

6 

SPEARS  Off-line  Training  Module 

CAT 

5 

7 

SPEARS  On-line  Testing/  Training 

CX 

f 

8 

SPEARS  On-line  Testing/  Training 

CAT 

6 

- ? - 

4.2.1  Developmental  Test  &  Evaluation 

DT&E  is  an  iterative  process  of  design,  build,  test,  analyze,  fix,  and  retest.  It  is  conducted  in 
a  controlled,  benign  environment  at  the  contractor’s  facilities  to  demonstrate  that  the 
engineering  design  and  development  process  is  complete.  DT&E  tests  for  compliance  to 
engineering  specification  and  ensures  that  the  product  is  ready  for  government  acceptance. 
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Six  unique  DT&E  activities  are  planned  for  this  RDT&E  program  to  ensure  that  the  prototype 
technique  or  device  meets  functional  requirements  and  represents  a  mature  design.  Table 
4.2. 1-1  identifies  the  issues  that  each  DT&E  activity  will  address  from  Section  2.3.2  as  well  as 
the  responsible  agent  for  test  conduct. 
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4.2.2  Operational  Test  &  Evaluation 

All  OT&E  on  this  program  is  also  known  as  the  Airport  Demonstrations.  Operational  testing 
has  two  phases,  initial,  and  follow-on,  which  will  assess  the  operational  effectiveness  of  the 
test  items.  The  primary  difference  between  initial  and  follow-on  is  their  scope.  Only  CX 
systems  will  undergo  both  OT&E  phases,  and  only  lOT&E  will  be  performed  on  CAT 
systems.  Though  there  are  14  separate  tests  spanning  OT&E,  the  issues  that  are  addressed  are 
the  same  and  are  shown  in  Table  4. 2. 2-1  with  the  organization  responsible  for  the  conduct  of 
the  test.  The  last  column  is  split  to  indicate  responsibility  for  initial,  baseline  and  follow-on 
testing. 


•  Initial  OT&E  (lOT&E)  is  the  first  test  phase  which  evaluates  the  test  item  under 
realistic  or  nearly  realistic  operational  conditions  at  operational  sites.  It  includes 
certified  screeners  and  FAA  test  baggage  and/or  passenger  baggage  in  real-time  as 
well  as  digitized  bag  images  off-line.  For  this  program,  both  CX  and  CAT  related 
systems  will  undergo  lOT&E,  but  only  CX  system  will  undergo  Baseline  Testing. 
Baseline  Tests  define  the  baseline  or  untreated  screener  performance  to  support 
comparative  analysis  and  evaluation.  All  Baseline  Testing  will  generally  occur  at 
the  same  time  with  an  expected  test  duration  of  one  week  or  less  per  airport.  The 
conduct  of  CAT  SPEARS  lOT&E  will  be  coincident  with  the  BIS-EDS 
Demonstration. 

•  Follow-on  OT&E  (FOT&E)  will  consist  of  deployment  and  testing  of  the  test 
items  at  a  larger  diversity  of  selected  Category  X  airport  sites  for  CX  systems 
only. 


29 


OT&E,  CAT  SPEARS  On-line  Testing/Training  COIC  #13  •  Insertion  Air 

COIC#14  •  Vigilance 

COIC  #15  •  Usability 

_ CTIC  #20-23  0  Feedback,  Security,  Image  Content,  Customization 


AvSec  HF  TEMP 
20  June  1995 


5.0  REFERENCED  DOCUMENTS 

a.  DOT/FAA/CT-95/31,  BIS  Test  and  Evaluation  Master  Plan,  Baggage  Inspection 
Systems  -  Airport  Operational  Demonstration  Project,  dated  January  1995. 

b.  DOT/FAA/CT-94/65,  SPEARS:  Human  Factors  Functional  Requirements  and 
Procedures. 

c.  GAO/RCED-93-38,  Jan  27,  1994,  Aviation  Security:  Additional  Actions  Needed  to 
Meet  Domestic  and  International  Challenges. 

d.  GAO/RCED-94-142,  May  19,  1994,  Aviation  Security:  Development  of  New  Security 
Technology  Has  Not  Met  Expectations. 

e.  Executive  Order  12686,  The  President’s  Commission  on  Aviation  Security  and 
Terrorism. 

f.  FAA/ACS  Mission  Need  Statement  #163  and  Research  Project  Directive  #127. 

g.  DOT/FAA/CT-94/1 12,  Test  and  Evaluation  Report  for  Improved  Explosive  Device 
Detection  System  (lEDDS)  by  J.L.  Fobes,  Ph.D.  and  Ronald  J.  Lofaro,  Ph.D., 
December  1994. 

h.  DOT/FAA/CT-94/85,  Test  and  Evaluation  Report  for  Explosive  Detection  Device 
(CTX  5000)  Alarm  Resolution  by  J.L.  Fobes,  Ph.D.  and  J.  Connelly,  Ph.D.,  October 
1994. 


